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An  Investigation  of  Sensory  Information,  Levels  of  Automation,  and 
Piloting  Experience  on  Unmanned  Aircraft  Pilot  Performance 


INTRODUCTION 

A  recent  review  of  sensory  information  in  Unmanned 
Aircraft  Systems  (UAS),  compared  to  manned  aircraft, 
documented  the  severe  lack  of  information  available  to 
UAS  pilots  (Williams,  2008).  The  current  research  con¬ 
tinues  that  review  to  find  empirical  support  for  the  need 
to  have  multiple  sources  of  sensory  information  available 
to  pilots  of  UAS  to  enhance  their  ability  to  diagnose  and 
respond  to  system  failures. 

Research  looking  at  (UAS)  accident  causal  factors  has 
suggested  that  sensory  deficiencies  have  played  a  role  in 
UAS  accidents  (Tvaryanas,  Thompson,  &  Constable, 
2005).  For  example,  Tvaryanas  et  al.  stated  that  10% 
of  UAS  accidents  across  all  services  were  influenced  by 
a  misperception  of  the  location  and/or  attitude  of  the 
aircraft.  Additionally,  they  found  that  26.5%  of  Predator 
UAS  accidents  they  reviewed  had  problems  associated 
with  the  instrumentation  and  sensory  feedback  systems. 

However,  other  factors  besides  the  types  of  sensory 
information  available  can  influence  the  ability  of  a  pilot 
to  effectively  manage  a  flight.  UAS  control,  for  many  cur¬ 
rent  systems,  is  highly  automated.  Automation-induced 
complacency,  which  is  the  tendency  for  humans  to  become 
less  vigilant  or  focused  on  a  task  that  is  being  performed 
by  automation,  is  possible  when  automation  replaces 
a  task  that  occupies  a  human  activity  (Parasuraman, 
1986;  Warm,  1984).  Automation-induced  complacency 
has  been  documented  as  a  cause  or  contributing  factor 
in  many  manned  aircraft  accidents  throughout  the  last 
two  decades  (Parasuraman,  Molloy,  &  Singh,  1993; 
Singh,  Molloy,  &  Parasuraman,  1993;  Woods,  Sarter, 
&  Billings,  1997).  A  pilot’s  ability  to  respond  to  system 
failures,  therefore,  will  be  influenced  not  only  by  the 
sensory  information  available  but  also  by  the  type  and 
level  of  automation  employed  in  the  system  and  the 
control-interface  requirements  on  the  pilot. 

Finally,  there  is  an  unresolved  question  regarding  the 
need  for  manned  aircraft  experience  for  piloting  a  UAS 
(Fogel,  Gill,  Mout,  Hulett,  &Englund,  1973;  Schreiber, 
Lyon,  Martin,  &  Confer,  2002;  Williams,  2007). Tvarya¬ 
nas  et  al.  (2005)  found  that,  for  other  systems  besides  the 
Predator,  the  presence  of  sensory  feedback  as  a  factor  in 
the  accidents  was  not  as  readily  apparent.  They  speculated 
that  the  difference  could  be  that  Air  Force  UAS  pilots 
are  more  likely  to  notice  the  lack  of  sensory  information 


available  in  the  ground  control  station  because  of  their 
greater  experience  with  manned  aircraft  operations.  UAS 
pilots  in  other  military  branches  are  not  required  to  have 
experience  flying  manned  aircraft.  Currently,  the  Federal 
Aviation  Administration  (FAA)  requires  pilots  of  UAS  to 
have  a  manned  aircraft  pilot  certificate  for  most  opera¬ 
tions.  However,  the  development  of  a  UAS-specific  pilot 
certification  has  been  proposed. 

The  current  experiment  was  intended  to  examine  the 
effect  of  sensory  information  on  pilot  reactions  to  system 
failures  within  a  UAS  control  station  simulation.  However, 
in  addition  to  sensory  information,  this  research  also 
investigated  the  level  of  automation  used  in  controlling 
the  aircraft  and  the  level  of  manned  flight  experience 
of  the  participants,  since  these  also  have  been  shown  to 
influence  pilot  effectiveness.  The  design  of  the  experi¬ 
ment  was  a  2x2x2  between-subjects  design,  manipulat¬ 
ing  two  levels  of  sensory  information  (visual  vs.  visual/ 
auditory),  two  levels  of  control  automation  (manual  vs. 
automatic),  and  two  levels  of  manned  piloting  experience 
(some  vs.  none).  Participants  were  asked  to  pilot  a  UAS 
along  a  predetermined  route  of  flight  while  responding 
to  various  system  failures.  They  had  to  monitor  traffic 
in  the  area  and,  at  set  times  during  the  flight,  determine 
the  relative  position  of  the  aircraft  to  a  specific  location. 
It  was  expected  that  the  visual/auditory  level  of  sensory 
information  would  be  superior  to  the  visual-only  level, 
and  that  participants  would  respond  to  system  failures 
more  quickly  when  they  received  both  a  visual  and  audi¬ 
tory  failure  cue.  For  the  two  levels  of  automation,  it  was 
expected  that  the  more  automated  condition  would  lead 
to  a  certain  level  of  complacency  for  the  participants,  thus 
inducing  slower  responses  to  system  failures  and  perhaps 
poorer  performance  at  monitoring  traffic.  Finally,  par¬ 
ticipants  with  manned-aircraft  experience  were  expected 
to  be  better  at  determining  the  relative  position  of  the 
aircraft  and,  because  of  a  more  effective  scan,  detecting 
system  failures  in  the  visual-only  condition. 

METHOD 

The  study  was  conducted  at  the  Federal  Aviation  Admin¬ 
istration  (FAA)  Civil  Aerospace  Medical  Institute  (CAMI), 
Human  Factors  Laboratory  in  Oklahoma  City,  OK. 
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Table  1.  Participant  demographics  by  level  of  flight  experience. 


Flight 

Experience 

Mean  Total  Flight 
Hours  (Median) 

Mean  Flight  Hours 
Last  90  Days 

Mean  Age 
(Median) 

Gender:  Males 
(Females) 

Non-pilot 

0 

0 

23.6  (23) 

7(9) 

Pilot 

675.2  (265) 

22.5 

28.4  (23) 

12(4) 

Figure  1.  UAS  control  station  simulation  used  in  the  experiment. 


Participants 

Thirty-two  participants  were  recruited  from  the  local 
Oklahoma  City  metropolitan  area.  Of  the  32  participants, 
1 6  had  flown  as  pilot-in-command  (PIC)  of  a  manned 
aircraft  and  held  at  least  a  private  pilot  certificate.  The 
lowest  level  of  manned  flight  experience  was  73  hours; 
the  highest  was  3,000  hours.  The  other  16  participants 
had  no  flight  experience.  All  participants  were  asked  to 
fill  out  a  demographics  questionnaire  prior  to  the  start  of 
the  experiment.  Table  1  summarizes  these  demographic 
data  for  each  of  the  two  groups. 

In  addition,  participants  were  asked  about  their  experi¬ 
ence  with  UAS,  radio-controlled  aircraft,  and  computer 
gaming.  None  of  the  participants  had  any  UAS  experience. 
Four  participants  (three  non-pilots,  one  pilot)  had  flown 
radio-controlled  aircraft.  Plowever,  the  most  experience 
for  any  one  participant  was  20  hours.  The  level  of  gam¬ 
ing  experience  was  relatively  low,  with  most  participants 
(23)  stating  that  they  played  games  either  “a  few  times  a 
month”  (14)  or  “none”  (9). 


Apparatus 

An  unmanned  aircraft  simulation  was  created  using 
Microsoft  Flight  Simulator  2004©  (MFS04)  as  the  flight 
model.  A  separate,  generic  UAS  pilot  control  station  was 
developed  using  a  commercial  software  package  that  allows 
flight  variables  to  be  read  from  and  written  to  MFS04. 
This  software  package  is  the  Flight  Simulator  Universal 
Inter-Process  Communication  (FSUIPC)  package. 

Using  the  Visual  Basic  6.0  software  development 
environment,  a  UAS  control  station  simulation  was 
developed  that  could  control  an  aircraft  within  MFS04 
through  the  FSUIPC  software  module.  Any  aircraft 
flight  model  that  can  be  hosted  within  MFS04  can  be 
controlled  using  the  UAS  simulation  if  the  aircraft  has 
an  autopilot.  Flowever,  for  this  experiment,  we  utilized 
a  Cessna  172  flight  model. 

Figure  1  presents  a  picture  of  the  control  station. 
The  control  station  simulation  provides  for  three  types 
of  aircraft  control.  Manual  control  can  be  accomplished 
through  use  of  the  throttle  and  joystick.  Vector  control  is 
done  using  the  mouse  and  onscreen  buttons  for  changing 
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the  altitude  and  heading  of  the  aircraft.  Waypoint  control 
is  accomplished  by  entering  a  series  of  waypoints  on  the 
moving-map  display  and  establishing  altitude  settings  for 
each  leg  of  the  flight.  For  this  experiment,  only  vector 
and  waypoint  control  schemes  were  used. 

The  ground  control  station  display  interface  for  the 
experiment  consisted  of  a  moving  map  display  that 
depicted  ownship,  flight  waypoints  and  segments,  and 
air  and  ground  traffic.  Two  check  boxes  were  available 
to  remove  the  air  traffic  and/or  ground  traffic  from  the 
display.  Several  boxes  contained  flight  parameters  and 
radio  settings;  two  boxes  indicated  the  commanded 
heading  and  altitude  of  the  aircraft.  There  was  also  a  box 
that  indicated  the  datalink  status  and  was  color-coded 
as  green,  yellow,  or  red,  with  red  indicating  a  failure  of 
the  datalink.  The  pilot/operator  could  change  aircraft 
heading  and  altitude  by  using  a  mouse  to  click  on  com¬ 
mand  buttons.  Heading  and  altitude  values  could  also  be 
entered  using  a  keyboard,  but  this  method  of  interaction 
was  not  shown  to  the  pilot/ operator  for  this  experiment. 

The  interface  indicated  the  current  flight  segment. 
When  the  aircraft  came  within  0.8  miles  of  the  next 
waypoint,  the  flight  segment  number  was  increased  by 
one.  When  the  aircraft  was  being  flown  by  waypoint 
control,  new  heading  and  altitude  values  were  automati¬ 
cally  placed  in  the  command  boxes  by  the  system.  Under 
vector  control,  the  segment  number  increase  prompted 
the  pilot  to  input  new  heading  and  altitude  values  in  the 
command  boxes.  Above  the  moving  map  display,  there 
was  a  box  labeled  “Flight Technical  Error,”  indicating  the 
distance  of  the  aircraft  from  the  current  flight  segment. 
An  increasing  value  in  the  box  meant  that  the  aircraft 
was  flying  away  from  the  segment,  and  a  decreasing  value 
indicated  the  aircraft  was  flying  toward  the  segment.  The 
user  could  place  the  cursor  on  an  object  in  the  moving 
map  display  and  two  boxes  depicted  the  distance  of  the 
object  from  ownship  and  the  bearing  to  that  object. 

There  were  potentially  four  types  of  failures  that  could 
be  introduced  by  the  experimenter  during  a  flight.  These 
were  a  loss  of  datalink,  an  altitude  control  failure,  a  head¬ 
ing  control  failure,  and  an  engine  failure.  For  the  current 
experiment,  only  two  of  the  four  failure  types  actually 
occurred.  These  were  the  heading  control  failure  and 
engine  failure.  Each  failure  was  accompanied  by  a  visual 
warning.  This  warning,  except  for  the  datalink  failure, 
was  that  the  appropriate  aircraft  parameter  readout 
numbers  (altitude,  heading,  or  engine  revolutions  per 
minute)  turned  red.  A  datalink  failure  was  indicated  by 
the  datalink  status  box  turning  red.  There  could  also  be 
an  auditory  alarm.  The  same  alarm  (a  continuous  beeping 
sound)  was  used  for  each  of  the  warnings.  In  addition, 
participants  could  also  hear  simulated  engine  noise  at  the 
discretion  of  the  experimenter.  Participants  were  given 


either  a  visual  warning  only  or  a  visual  warning  accom¬ 
panied  by  an  auditory  alarm.  In  addition,  participants 
receiving  the  auditory  alarm  could  also  hear  engine  noise 
during  the  experiment. 

For  each  failure,  the  proper  response  for  the  failure 
was  to  press  a  corresponding  “recovery”  button  located 
below  each  flight  parameter  box  (in  the  case  of  altitude, 
heading,  or  engine  failures)  or  to  the  right  of  the  datalink 
status  box.  After  pressing  one  of  the  “recovery”  buttons,  a 
“recovering”  message  was  displayed  below  the  button  to 
indicate  that  the  system  was  recovering,  even  though  the 
alarm  might  still  be  activated.  The  “recovering”  message 
was  needed  because  the  system  required  several  seconds 
to  reduce  the  difference  between  the  commanded  head¬ 
ing  or  altitude  and  the  actual  heading  or  altitude.  When 
heading  or  altitude  changes  were  commanded,  the  alarm 
could  be  triggered,  but  the  user  was  instructed  that  the 
presence  of  the  “recovering”  message  indicated  that  no 
failure  had  occurred.  Because  of  the  trivial  nature  of 
failure  recovery,  this  behavior  was  thought  to  add  some 
complexity  to  the  interface  to  make  it  more  consistent 
with  actual  control  station  complexity. 

Materials 

Several  items  were  developed  for  the  experiment.  A 
participant  consent  form  was  constructed  explaining 
the  experiment  and  the  requirement  to  remain  seated 
for  approximately  1  hour  during  the  experiment.  An 
experimenter  instruction  page  was  developed  to  read  to 
the  participants.  A  demographics  questionnaire  was  cre¬ 
ated  to  gather  information  regarding  age,  gender,  flight 
experience,  unmanned  aircraft  experience,  radio-control 
aircraft  experience,  and  computer-gaming  experience. 
The  experimenter  instruction  page  and  demographics 
questionnaire  are  reproduced  in  Appendix  A.  A  page 
with  an  analog  clock  face  was  used  to  explain  the  desig¬ 
nation  of  relative  position  information.  For  non-pilots, 
a  heading  indicator  depiction  was  used  to  explain  the 
concept  of  heading  and  what  it  meant  to  fly  a  heading. 
An  experimenter  checklist  was  developed  to  track  experi¬ 
ment  setup  and  conduct,  including  the  introduction  and 
order  of  failures,  relative  position  responses,  and  sighting 
of  traffic.  Finally,  a  post-test  questionnaire  was  created 
to  gather  workload  estimates,  subjective  impressions  of 
the  display  interface,  and  participant  comments  regard¬ 
ing  the  experiment.  This  questionnaire  can  be  found  in 
Appendix  B. 

Procedures 

Participants  were  tested  one  at  a  time.  Each  partici¬ 
pant  was  brought  into  the  simulation  lab  and  asked  to 
read  and  sign  the  consent  form.  They  were  then  asked 
to  fill  out  the  demographics  questionnaire.  After  com- 
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pleting  the  questionnaire,  participants  were  read  a  set  of 
instructions  regarding  the  experiment.  Participants  were 
told  they  would  be  the  pilot  of  an  unmanned  aircraft. 
In  the  automated  flight  condition,  their  task  would  be 
to  simply  monitor  the  aircraft  and  potentially  respond 
to  failures.  In  the  vector-control  condition,  their  task 
would  require  making  heading  and  altitude  changes  to 
the  aircraft  during  the  flight  to  maintain  its  progress 
along  a  predetermined  route  of  flight  and  potentially 
respond  to  failures.  Participants  then  received  training 
regarding  the  control  station  interface  and  were  briefed 
about  potential  failures  and  the  appropriate  responses 
to  those  failures.  They  were  also  told  that  they  would  be 
asked  to  indicate  the  relative  position  of  locations  from 
the  aircraft.  Finally,  they  were  instructed  to  watch  for 
air  traffic  and  to  call  out  traffic  that  came  within  5  miles 
horizontally  and  1 ,000  feet  vertically  of  their  aircraft. 

After  receiving  their  instructions,  participants  practiced 
a  flight  scenario.  The  practice  scenario  consisted  of  a  flight 
of  three  waypoints,  requiring  one  turn  by  the  aircraft. 
Any  confusion  regarding  the  task  or  interaction  with  the 
control  station  was  corrected,  and  the  participants  then 
flew  the  experimental  scenario.  The  experimental  scenario 
was  a  flight  of  four  waypoints  (three  flight  legs) .  The  total 
length  of  the  flight  was  approximately  60  miles  and  took 
approximately  40  minutes  to  complete.  Both  the  practice 
and  experimental  scenarios  began  with  the  aircraft  in  the 
air  and  ended  before  reaching  the  final  waypoint. 

During  the  course  of  the  experimental  scenario,  four 
failures  were  introduced.  Two  of  the  failures  occurred  dur¬ 
ing  the  second  flight  leg,  and  two  failures  occurred  during 
the  third  flight  leg.  The  only  failures  that  occurred  were  a 
heading  control  failure  and  an  engine  failure.  The  order 
of  the  failures  was  counterbalanced  across  participants 
and  conditions.  Between  each  failure  on  each  leg,  the 
participant  was  asked  to  estimate  the  relative  direction 
of  the  starting  point  of  the  flight. 

After  completing  the  experimental  scenario,  partici¬ 
pants  were  asked  to  fill  out  a  workload  estimate  based 
on  the  NASA  TLX  subjective  workload  scale  (Hart  & 
Staveland,  1988).  NASATLX  is  a  six- item  questionnaire 
measuring  mental  demand,  physical  demand,  temporal 
pressure,  perceived  performance,  total  mental  and  physi¬ 
cal  effort,  and  frustration  level.  Participants  scored  their 
workload  on  two  tasks,  piloting  the  aircraft  and  responding 
to  failures.  Each  scale  was  scored  on  a  seven-point  metric 
by  marking  a  point  along  a  line  representing  each  scale. 
Participants  also  scored  their  subjective  assessment  of 
the  complexity  of  the  user  interface  and  the  difficulty  of 
identifying  a  failure  along  the  same  seven-point  metric. 
Participants  were  debriefed  and  released  upon  completing 
the  questionnaire. 


RESULTS 

Responding  to  Failures 

Two  significant  findings  appeared  in  regard  to  respond¬ 
ing  to  failures.  First,  the  proportion  of  participants  that 
failed  to  respond  to  an  engine  failure  within  5  seconds 
was  significantly  greater  when  no  sounds  (auditory  alarm 
or  engine  noise)  were  present  in  the  control  station, 
X2  =  4.57,p=.033.  Twenty- five  percent  of  the  participants 
failed  to  respond  within  the  5-second  interval  when  no 
sounds  were  present  in  the  control  station  as  opposed  to 
none  when  sounds  (both  auditory  alarm  and  engine  noise) 
were  present.  Neither  the  level  of  automation  nor  level 
of  pilot  experience  had  any  significant  effect  on  response 
to  an  engine  failure. 

The  second  significant  finding  was  that  the  proportion 
of  participants  responding  to  a  heading  control  failure 
before  the  visual  failure  warning  occurred  was  significantly 
greater  if  the  participant  was  an  experienced  pilot  and 
no  auditory  warning  was  present,  y2  =  9.93,  p  =  .019. 
Three  of  eight  pilots  in  the  no-sound  condition  (38%) 
pushed  the  heading  failure  recovery  button  before  the 
failure  warning  had  occurred,  as  opposed  to  zero  pilots 
in  the  sound  condition,  and  zero  non-pilots  in  either  the 
sound  or  no-sound  conditions. 

In  regard  to  responding  to  heading  control  failures, 
participants  received  the  following  instructions:  “If  the 
heading  control  fails  during  the  flight,  the  actual  heading 
will  begin  to  drift  away  from  the  commanded  heading. 
Once  the  actual  heading  and  commanded  heading  dif¬ 
fer  by  10  degrees  or  more,  the  actual  heading  numbers 
will  turn  red  and  an  alarm  will  sound.”  Note  that  the 
auditory  alarm  instructions  were  provided  only  to  those 
participants  who  were  given  both  a  visual  and  auditory 
failure  warning. 

Flight  Control  Accuracy 

The  initial  heading  of  the  aircraft  at  the  beginning  of 
the  experimental  scenario  was  approximately  3  degrees 
left  of  the  heading  required  to  fly  directly  to  the  second 
waypoint.  If  the  pilot/ operator  did  not  make  corrections 
to  the  heading,  the  aircraft  would  miss  the  second  way- 
point  by  approximately  0.8  miles.  Somewhat  unexpect¬ 
edly,  several  of  the  non-pilot  participants  did  not  make 
any  corrections  to  the  heading  of  the  aircraft  during  the 
initial  leg  of  the  flight.  However,  all  of  the  pilots  made 
the  appropriate  corrections.  This  led  to  a  finding  that 
the  technical  flight  error  for  non-pilots  was  much  higher 
than  the  technical  flight  error  for  pilots  during  the  first 
leg  of  the  flight,  for  those  participants  in  the  manual 
flight  condition.  Of  course,  for  those  participants  in  the 
automated  flight  condition,  appropriate  corrections  to 
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Figure  2.  Flight  technical  error  during  the  first  flight  leg. 


the  heading  were  accomplished  by  the  system.  These  dif¬ 
ferences  can  be  seen  in  a  plot  of  the  flight  technical  error 
measures  for  the  first  flight  leg  (Figure  2). 

An  analysis  of  variance  of  these  data  demonstrated  a 
significant  automation  level  by  experience  level  interac¬ 
tion,  F(l,24)  =  7.892,  p  =  .01.  In  addition,  both  the 
main  effects  for  automation  level  and  for  pilot  experience 
level  were  significant,  F(l,  24)  =  29.24,  p  <  .001,  and 
F(l,  24)  =  8.255 , p  =  .008,  respectively.  As  can  be  seen 
in  Figure  2,  automated  flight  control  was  more  accurate 
than  manual  flight  control.  In  addition,  for  the  manual 
flight  control  condition,  pilot  flight  accuracy  was  better 
than  non-pilot  flight  accuracy. 

A  similar  finding  appeared  for  the  second  leg  of  the 
flight  as  well.  Several  of  the  non-pilots  did  not  make 
corrections  to  the  heading  of  the  aircraft  after  entering 
an  initial  heading  change  at  the  waypoint.  In  addition, 
after  the  heading  control  failure,  and  once  the  aircraft 
had  recovered  its  commanded  heading,  many  non-pilots 
made  no  corrections  to  account  for  the  aircraft  straying 
off  course.  None  of  the  pilots  exhibited  this  behavior. 

A  fairly  simplistic  algorithm  was  used  for  the  automated 
flight  mode.  This  algorithm  triggered  a  turn  to  a  new 
waypoint  when  the  aircraft  got  to  within  0.8  miles  of  the 
previous  waypoint.  For  the  second  leg,  because  the  turn 
to  the  leg  was  shallow,  the  automated  flight  mode  was 


less  accurate  than  the  first  leg.  As  can  be  seen  in  Figure 
3,  pilots  flying  manually  were  more  accurate  than  the 
automated  flight  mode  for  the  second  leg.  An  analysis 
of  variance  of  the  flight  technical  error  for  the  second  leg 
found  a  significant  automation  level  by  experience  level 
interaction,  F(l,  24)  =  4.932,  p  =  .036.  The  main  effect 
for  pilot  experience  level  was  also  significant,  F(l,  24)  = 
4.638 ,p  =  .042.  Flowever,  the  main  effect  for  automation 
level  was  not  significant. 

As  can  be  seen  in  Figure  3,  pilots  flying  in  manual 
mode  were  significantly  more  accurate  than  the  non¬ 
pilots.  Of  course,  there  was  no  difference  between  pilots 
and  non-pilots  in  automatic  mode.  This  accounts  for  the 
significant  interaction  effect  that  was  found. 

Interacting  With  the  Display 

Although  display  interaction  was  limited  for  all  par¬ 
ticipants,  especially  those  in  the  automated  condition, 
it  was  possible  to  change  the  zoom  level  of  the  moving- 
map  display  during  the  flight.  For  each  participant,  an 
assessment  was  made  of  the  level  of  zoom  manipulation 
performed  (high  or  low).  Low  levels  of  zoom  manipula¬ 
tion  meant  that  the  zoom  level  was  rarely  changed  during 
the  flight  or  not  at  all.  Fligh  levels  of  zoom  manipulation 
occurred  usually  when  the  participant  was  attempting 
to  fly  the  aircraft  as  close  to  the  flight  path  as  possible. 
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Figure  3.  Flight  technical  error  during  the  second  flight  leg. 


Figure  4.  Number  of  participants  with  high  levels  of  zoom  manipulation. 


Results,  seen  in  Figure  4,  show  that  pilots  in  the  manual 
condition  tended  to  manipulate  the  zoom  more  than 
participants  in  the  other  conditions,  %2  =11 .73,  p  =  .008. 
This  higher  level  of  zoom  manipulation  was  reflected  in 
the  lower  flight  technical  error  scores  discussed  above. 

Monitoring  Traffic 

No  statistical  differences  in  the  ability  to  monitor 
traffic  were  found.  One  pilot  and  one  non-pilot  failed 
to  call  out  traffic  that  came  within  5  miles  horizontally 
and  1,000  feet  vertically  of  the  aircraft.  In  addition,  one 
pilot  and  two  non-pilots  called  out  traffic  that  was  out¬ 


side  the  prescribed  parameters.  The  pilot  that  failed  to 
call  out  the  traffic  had  set  the  zoom  level  of  the  moving 
map  display  to  1  mile  (while  attempting  to  reduce  flight 
technical  error)  and  failed  to  see  the  impinging  traffic. 
The  non-pilot,  on  the  other  hand,  had  the  traffic  visible 
on  the  moving-map  display  but  was  apparently  focused 
on  a  different  portion  of  the  screen. 

Awareness  of  Relative  Position 

No  statistical  differences  in  awareness  of  relative  posi¬ 
tion  were  found.  Participants,  in  general,  understood  the 
concept  of  “o’clock”  position  and  could  accurately  estimate 
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the  direction  to  a  specific  location  (starting  waypoint) 
from  the  current  position  and  heading  of  the  aircraft. 

Workload  and  Other  Subjective  Estimates 

After  completion  of  the  flight,  participants  were  asked 
to  provide  subjective  estimates  of  workload  related  to 
piloting  the  UAS  and  responding  to  failures.  In  addition, 
they  were  asked  the  questions,  “How  complex  was  the 
user  interface?”  and  “How  difficult  was  it  to  identify  a 
failure?”  Responses  were  along  a  seven-point  scale,  with 
higher  numbers  indicating  higher  workload,  complexity, 
or  difficulty.  Because  the  task  was  fairly  simple,  it  was 
expected  that  many  of  the  six  workload  scales  used  for 
the  NASA  TLX  would  not  yield  any  differences  across 
conditions.  One  exception  was  the  scale  for  temporal 
demand.  The  description  for  the  temporal  demand  rating 
scale  is,  “How  much  time  pressure  did  you  feel  due  to  the 
rate  or  pace  at  which  the  tasks  or  task  elements  occurred? 
Was  the  pace  slow  and  leisurely  or  rapid  and  frantic?” 

Results  showed  a  significant  difference  in  temporal 
demand,  both  in  piloting  the  UAS  and  in  responding 
to  failures  due  to  a  difference  in  the  level  of  automation, 
F(l,  24)  =  6.744,  p  =  .016  and  F(l,  24)  =  5.825,  p  = 
.024,  respectively.  No  other  main  effects  or  interactions 
reached  significance. 

An  analysis  of  the  responses  to  the  question,  “How 
difficult  was  it  to  identify  a  failure?”  showed  no  significant 
differences  for  any  of  the  factors  or  their  interactions. 
However,  an  interesting  pattern  of  responses  was  found 
for  the  question,  “How  complex  was  the  user  interface?” 
Both  the  main  effects  of  type  of  sensory  information  and 
pilot  experience  level  were  significant,  F(l,  24)  =  9.035, 
p  =  .006  and  F(l,  24)  =  4.271,  p  =  .050,  respectively. 
That  is,  participants  almost  always  judged  the  interface 
to  be  higher  in  complexity  when  there  was  no  sound 
available.  The  interface  was  judged  to  be  less  complex 
when  sound,  including  engine  noise,  was  available.  Also, 
non-pilots  judged  the  interface  to  be  more  complex  than 
pilots  under  most  conditions.  There  was  a  significant 
experience  level  by  sensory  information  interaction,  F  ( 1 , 
24)  =  4.271,p  =  .050.  Finally,  there  was  also  a  significant 
three-way  interaction  as  well  with  F(l,  24)  =  5.965 ,  p  = 
.022.  Figures  5a  and  5b  show  this  interaction. 

A  summary  of  this  three-way  interaction  effect  is  as 
follows.  Comparing  Figure5a  and  5b,  we  see  that  pilots 
generally  rated  the  interface  as  less  complex  than  did  non¬ 
pilots,  and  that  the  presence  of  sound  also  reduced  the 
complexity,  except  in  the  automated  condition  where  the 
presence  of  sound  reduced  complexity  only  for  the  non¬ 
pilots.  The  presence  or  absence  of  sound  had  little  effect  on 
ratings  of  complexity  for  pilots  in  the  automated  condition. 


DISCUSSION 

Flight  simulation  experiments  quite  often  introduce 
potentially  confounding  variables  by  requiring  pilot 
actions  and  decisions  beyond  what  is  of  interest  in  the 
experiment.  This  is  especially  true  with  simulations  that 
duplicate  the  control  interfaces  of  specific  systems.  Be¬ 
cause  of  this  lack  of  control,  it  is  difficult  to  assign  causal 
statements  to  many  of  the  results. 

The  notion  that  simply  adding  a  second  type  of  sensory 
information  (sound)  would  increase  the  ability  of  pilots  to 
identify  and  respond  to  failures  was  not  supported  in  the 
current  study.  While  the  presence  of  sound  did  improve 
responses  to  engine  failures,  it  did  not  improve  responses 
to  failures  in  heading  control.  One  difference  between  the 
engine  failure  cues  and  heading  control  failure  cues  was 
the  presence,  in  the  condition  where  sound  was  used,  of 
engine  noise  in  addition  to  the  auditory  warning.  Un¬ 
fortunately,  it  is  not  possible  to  determine  whether  this 
additional  sound  cue  was  the  cause  of  the  difference  in 
responding  to  the  failures.  It  is  possible  that  one  reason 
that  differences  in  sensory  information  did  not  affect  the 
ability  to  identify  heading  control  failures  is  that  much 
of  the  task  involved  making  heading  changes,  at  least 
in  the  manual  condition,  so  participants  were  more  fo¬ 
cused  on  the  heading  information  and  more  aware  when 
the  commanded  and  actual  heading  started  to  diverge. 
Nevertheless,  the  results  of  the  experiment  support  the 
long- accepted  notion  that  multiple  modes  of  informa¬ 
tion  are  more  effective  than  a  single  mode,  especially  for 
flight  parameters  that  are  not  monitored  constantly  (e.g. , 
Kantowitz  &  Sorkin,  1983). 

In  addition  to  its  effect  on  responses  to  system  failures, 
the  presence  of  sound  also  had  an  effect  on  the  subjec¬ 
tive  estimates  of  the  complexity  of  the  user  interface.  In 
general,  participants  had  the  impression  that  the  interface 
was  more  complex  when  no  sound  was  available.  The 
exception  was  pilots  in  the  automated  condition,  where 
the  presence  or  absence  of  sound  had  no  impact  on  their 
perceptions  of  complexity.  Why  this  would  affect  percep¬ 
tions  of  user-interface  complexity  is  not  clear.  However, 
it  could  be  that  participants  perceived  the  overall  task  to 
be  more  easily  accomplished  with  the  presence  of  sound, 
even  though  objective  measures  of  performance  only 
supported  the  response  to  engine  failures. 

The  prediction  that  higher  levels  of  automation  would 
lead  to  complacency  or  vigilance  decrements,  that  is,  a 
lack  of  focus  on  the  task,  was  not  supported  in  the  cur¬ 
rent  experiment.  Perhaps  the  relatively  short  flight  used 
for  the  experiment  (approximately  40  minutes)  did  not 
allow  for  an  effect  to  occur.  However,  Parasuraman  et 
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Figure  5a.  Complexity  rating  as  a  function  of  experience  level  and  sensory  information 
for  the  automated  condition. 


Figure  5b.  Complexity  rating  as  a  function  of  experience  level  and  sensory  information 
for  the  manual  condition. 


al.  (1993)  suggest  that  complacency,  at  least,  is  not  a 
function  of  time  as  much  as  it  is  a  function  of  variability 
in  automation  reliability.  In  that  study,  complacency  ef¬ 
fects  were  evident  after  only  20  minutes  of  monitoring. 
Constant  levels  of  automation  reliability  were  shown  by 
Parasuraman  et  al.  to  lead  to  higher  levels  of  complacency. 
However,  a  constant  level  of  automation  reliability  in  the 
current  experiment  did  not  result  in  complacency  effects. 
Perhaps  the  relatively  simple  nature  of  the  task  attenuated 
any  effects  due  to  differences  in  automation. 

Although  the  predicted  effects  of  automation  in  re¬ 
gard  to  complacency  or  vigilance  decrements  were  not 
supported,  other  effects  of  differing  levels  of  automation 
occurred.  As  expected,  a  higher  level  of  automation  led 
to  lower  estimates  of  subjective  workload.  This  was  re¬ 
flected  in  the  flight-technical-error  performance  findings 
that  showed  superior  flight  performance,  in  general,  for 
participants  in  the  high-automation  condition. 

The  manipulation  of  manned  aircraft  experience  lev¬ 
els  resulted  in  some  significant  findings  but  not  for  the 
tasks  where  differences  were  expected.  Participants  with 
manned  aircraft  experience  did  not  perform  any  better  at 
monitoring  traffic  or  in  estimating  relative  direction  than 
those  without  flight  experience.  However,  there  were  some 
interesting  differences  between  pilots  and  non-pilots  that 
bear  closer  examination  in  the  performance  of  the  tasks. 

The  finding  that  pilots,  in  the  manual  conditions,  flew 
significantly  closer  to  the  flight  path  than  non-pilots  was 
unexpected.  It  is  difficult  to  believe  that  only  the  pilots 
noticed  that  the  aircraft  was  deviating  from  the  flight  path 
during  the  first  flight  segment,  so  the  question  is  why  some 
of  the  non-pilots  did  not  attempt  to  correct  the  devia¬ 
tion.  Because  it  occurred  suggests  individual  differences 
between  the  pilots  and  some  of  the  non-pilots  could  be 
due  to  either  training  or  are  innate  traits  that  contribute 
to  success  as  a  pilot.  If  manned  aircraft  training  and/or 
experience  leads  to  more  responsive  flight-path  control, 
it  would  be  important  to  identify  what  portion  of  the 
training  was  responsible.  Interaction  with  onboard  auto¬ 
mated  systems  might  have  contributed.  Unfortunately, 
pilots  were  not  questioned  regarding  their  experience 
with  onboard  automation.  It  would  be  interesting  to  see 
if  the  addition  of  a  more  manual  mode  of  flight,  such  as 
joystick  control,  would  still  show  these  pilot/non-pilot 
differences. 


One  reviewer  has  suggested  that  the  non-pilots  were 
less  responsive  to  flight  path  control  because  of  their  low 
level  of  gaming  experience.  However,  the  pilots  also  re¬ 
ported  low  levels  of  gaming  experience.  Even  so,  if  a  lack 
of  gaming  experience  contributed  to  the  behavior,  there  is 
still  a  question  regarding  how  such  experience  is  similar 
to  manned  flight  training.  The  question  is  important 
in  regard  to  whether  or  not  manned  aircraft  training  is 
important  for  the  piloting  of  unmanned  aircraft. 

A  second  unexpected  difference  between  pilots  and 
non-pilots  in  the  study  was  in  the  response  to  heading- 
control  failures.  A  significant  proportion  of  pilots 
responded  to  the  heading  failure  before  the  warning 
for  the  failure  was  presented,  based  on  a  recognition  of 
the  actual  heading  drifting  away  from  the  commanded 
heading.  However,  this  occurred  only  in  the  no-sound 
condition.  The  presence  of  an  auditory  warning  for  pilots 
actually  seemed  to  inhibit  a  response  to  a  heading  failure. 
None  of  the  non-pilots  responded  early  to  the  heading 
control  failure,  regardless  of  the  warning  condition.  For 
both  pilots  and  non-pilots,  it  was  clear  that  some  of 
the  participants  noticed  the  heading  failure  early  but 
waited  for  the  warning  by  positioning  the  cursor  over  the 
heading  recovery  button.  Again,  there  are  questions  of 
whether  individual  differences  allowed  some  of  the  pilots 
to  respond  early,  why  the  presence  of  a  sound  cue  would 
prevent  this  response,  and  whether  training  or  other  fac¬ 
tors  were  involved  in  the  differences  between  the  groups. 

In  conclusion,  the  presence  of  sound  cues  in  addition 
to  visual  cues  was  helpful  for  the  recognition  of  engine 
failures,  but  auditory  warnings  were  not  necessarily  useful 
for  information  that  was  constantly  being  monitored  by 
the  pilot,  like  aircraft  heading.  In  addition,  more  research  is 
required  in  regard  to  the  question  of  the  effect  of  manned 
aircraft  experience  on  the  piloting  of  unmanned  aircraft. 
The  results  of  the  current  experiment  suggest  differences 
between  those  with  manned  aircraft  experience  and  those 
without,  but  it  is  unclear  whether  these  differences  are 
due  to  manned  aircraft  training  and  flight  experience 
or  whether  other  factors,  such  as  personality,  may  be 
evident.  Identifying  the  cause  of  these  differences  could 
affect  training  and/or  selection  requirements  for  pilots 
of  unmanned  aircraft  systems. 
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Appendix  A 

Subject  Instructions  for  UAS  Experiment 


Welcome.  In  this  study  you  will  be  asked  to  be  the  pilot  of  an  unmanned  aircraft  system. 
Your  task  is  to  monitor  the  flight  and,  unless  the  flight  control  is  automated,  make 
heading  and  altitude  changes  to  the  aircraft  during  the  flight  to  maintain  its  progress 
along  a  predetennined  route  of  flight.  Monitoring  the  flight  will  consist  of  tracking 
heading,  altitude,  engine  rpm,  and  datalink  status  of  the  aircraft.  There  are  potentially 
four  types  of  failures  that  could  occur  during  the  flight.  If  the  datalink  is  lost  for  more 
than  5  seconds,  an  alarm  will  sound.  If  the  altitude  control  fails  during  the  flight  and  the 
altitude  varies  from  the  commanded  altitude  by  more  than  450  feet,  an  alarm  will  sound 
and  the  altitude  readout  numbers  will  turn  red.  If  the  heading  control  fails  during  the 
flight  and  the  heading  varies  from  the  commanded  heading  by  more  than  10  degrees,  an 
alarm  will  sound  and  the  heading  readout  number  will  turn  red.  If  the  engine  power  fails 
during  the  flight  and  the  engine  rpm  drops  below  2000,  an  alarm  will  sound  and  the  RPM 
readout  number  will  turn  red.  For  each  of  these  failures,  the  proper  response  for  that 
failure  will  be  to  press  the  corresponding  “recovery”  button. 


In  addition  to  this  task,  the  experimenter  will  ask  you  to  estimate  your  relative  position  to 
a  specific  location  at  various  times  during  the  flight.  Your  response  to  this  request  should 
be  an  “o’clock”  position  indicating  the  relative  position  from  the  aircraft  to  that  location. 
To  properly  answer  the  question,  you  must  take  into  account  the  direction  that  the  aircraft 
is  flying  in  addition  to  the  location  of  the  position.  If  the  location  is  directly  in  front  of  the 
aircraft,  the  answer  would  be  “12  o’clock.”  If  the  location  is  to  the  right  of  the  aircraft’s 
path  of  flight,  the  correct  answer  would  be  “3  o’clock.”  In  each  case  the  answer  would  be 
relative  to  someone  actually  flying  in  the  aircraft.  This  task  can  be  confusing  because  if 
the  aircraft  is  flying  south,  left  and  right  of  the  aircraft  are  reversed  from  someone 
looking  at  it  from  the  outside.  Keep  that  in  mind  as  you  answer  the  question. 

Questions?  If  not,  let’s  take  a  look  at  the  control  interface,  and  I  will  explain  the  details  of 
what  you  will  see  on  the  screen. 


At 


Appendix  A  (continued) 
Demographics  Questionnaire 


Participant  Number _  Date _ 

Instructions:  Indicate  your  response  to  the  following  questions  by  filling  in  the  blank  or  checking 
the  circle  or  square  corresponding  to  the  response  option  of  your  choice. 

FLIGHT  EXPERIENCE 

1.  Have  you  ever  piloted  an  unmanned  aircraft  system  (excluding  radio-control  hobby  aircraft)? 

_ Yes  _ No 

2.  If  Yes,  what  systems  have  you  flown  (please  list) 


3.  If  Yes,  what  is  your  total  flight  time  as  pilot  of  unmanned  aircraft  in  hours  approximately? 


4.  Do  you  have  any  experience  as  pilot-in-command  of  a  (manned)  aircraft?  _ Yes  _ No 

If  no,  please  skip  to  the  next  section  on  radio-controlled  aircraft  (RC)  experience. 

5.  How  long  have  you  been  a  certificated  pilot? _ years 

6.  What  is  the  highest  certificate  you  hold? 

•  Student  • 

•  Sport  • 

•  Recreational  • 

7.  What  ratings  do  you  hold?  [Mark  all  that  apply] 

•  Single-engine  Land  or  Sea  • 

•  Multi-engine  Land  or  Sea  • 

•  Rotorcraft  (helicopter/gyroplane)  • 

•  Lighter  than  air 

8.  Please  list  your  Pilot-in-Command  (PIC)  time 
hour.) 

Last  90  Days 

All  aircraft .  . Hrs 


RADIO-CONTROLLED  AIRCRAFT  EXPERIENCE 

9.  Have  you  ever  flown  a  radio-controlled  aircraft? _ Yes  _ No 

10.  If  yes,  how  many  hours  of  flight  experience  (approximately)  do  you  have? 


(Estimate  and  round  to  the  nearest  whole 

PIC  Flight  Time  (Hours) 

Instrument  Total  PIC 

_ Hrs  _ Hrs 


Private 

Commercial 

Airline  Transport  (ATP) 


Flight  Instructor  (CFI,  CFII,  MEI) 
Instrument 

Other  (please  specify) 
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Appendix  A  (continued) 


GAMING  EXPERIENCE 

1 1 .  How  often  do  you  play  computer  games? _ every  day  _ few  times  a  week _ few 

times  a  month _ none 

1 2.  What  types  of  games  do  you  play  (check  all  that  apply) _ first  person  shooter _ role 

playing 

_ word  or  board  games  _ Other  (please  specify) 


13.  Age 


14.  Gender _ M  _ F 
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Appendix  B 
Post-test  Questionnaire 

Task  -  Piloting  the  UAS 


MENTAL  DEMAND 

_ I _ I _ I _ I _ 

Low  High 

PHYSICAL  DEMAND 

_ I _ I _ I _ I _ 

Low  High 

TEMPORAL  DEMAND 

_ I _ I _ I _ I _ 

Low  High 

PERFORMANCE 

_ I _ I _ I _ I _ 

Good  Poor 


EFFORT 


Low  High 

FRUSTRATION 


Low  High 
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Appendix  B  (continued) 


Task  -  Responding  to  Failures 

MENTAL  DEMAND 


Low  High 

PHYSICAL  DEMAND 


Low  High 

TEMPORAL  DEMAND 


Low  High 

PERFORMANCE 


Good  Poor 


EFFORT 


Low  High 

FRUSTRATION 


Low  High 
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Appendix  B  (continued) 


Rating  Scale  Definitions 


Title 

Endpoints 

Descriptions 

MENTAL 

DEMAND 

LOW/HIGH 

How  much  mental  and  perceptual  activity  was 
required  (e.g.,  thinking,  deciding,  calculating, 
remembering,  looking,  searching,  etc.)?  Was 
the  task  easy  or  demanding,  simple  or 
complex,  exacting  or  forgiving? 

PHYSICAL 

DEMAND 

LOW/HIGH 

How  much  physical  activity  was  required 
(e.g.,  pushing,  pulling,  turning,  controlling, 
activating,  etc.)?  Was  the  task  easy  or 
demanding,  slow  or  brisk,  slack  or  strenuous, 
restful  or  laborious? 

TEMPORAL 

DEMAND 

LOW/HIGH 

How  much  time  pressure  did  you  feel  due  to 
the  rate  or  pace  at  which  the  tasks  or  task 
elements  occurred?  Was  the  pace  slow  and 
leisurely  or  rapid  and  frantic? 

PERFORMANCE 

GOOD/POOR 

How  successful  do  you  think  you  were  in 
accomplishing  the  goals  of  the  task  set  by  the 
experimenter  (or  yourself)?  How  satisfied 
were  you  with  your  performance  in 
accomplishing  these  goals? 

EFFORT 

LOW/HIGH 

How  hard  did  you  have  to  work  (mentally  and 
physically)  to  accomplish  your  level  of 
performance? 

FRUSTRATION 

LEVEL 

LOW/HIGH 

How  insecure,  discouraged,  irritated,  stressed 
and  annoyed  versus  secure,  gratified,  content, 
relaxed  and  complacent  did  you  feel  during 
the  task? 
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Appendix  B  (continued) 


How  complex  was  the  user  interface? 

_ I _ I _ _ I _ I _ 

Low  High 

How  difficult  was  it  to  identify  a  failure? 

_ I _ I _ _ I _ I _ 

Easy 

Difficult 

Comments? 


Thanks  for  your  participation. 
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